feat(eval): add measurement manifests for online scoring - #722

• Add default per-agent measurement manifests for managed-job online scoring.
• Enable trace_fitness (em-001 v1) for six stock agents; omit others (e.g., scribe).
• Document manifest ownership, resolution order, and separation from functional eval harness.

graph TD
  J(["Managed agent job"]) --> CLI["fullsend eval-measure"]
  CLI --> D{"Local manifest?"}
  D -->|"yes"| L["FULLSEND_DIR manifest"]
  D -->|"no"| R["agents@v0 manifest"]
  CLI --> T[("run-telemetry.jsonl")] --> S["trace_fitness scorer\n(em-001 v1)"] --> O[("Measurement results")]

  subgraph Legend
    direction LR
    _proc["Process/CLI"] ~~~ _dec{"Decision"} ~~~ _data[("Data")]
  end

Loading

The following are alternative approaches to this PR:

1. Bake defaults into fullsend (no agents@v0 fetch)

• ➕ Single release artifact controls both scorer code and default policy
• ➕ No reliance on agents@v0 pin being updated to enable scoring
• ➖ Harder for agents repo to own/iterate on policy independently
• ➖ Forces fullsend release for simple enable/disable/threshold policy changes

2. Single manifest with per-agent sections

• ➕ Fewer files to manage/version together
• ➕ Easier to diff global policy changes across agents
• ➖ Makes per-agent overrides less discoverable and more conflict-prone
• ➖ Resolution logic becomes more complex (merge/selection vs direct fetch)

3. Remote registry service for measurement policy

• ➕ Policy updates without Git pins; central governance and rollout controls
• ➕ Can support targeting/experiments by org/repo
• ➖ Adds new infrastructure dependency and failure modes
• ➖ Overkill for initial ship; higher operational complexity

Recommendation: Keep the current approach: per-agent YAML manifests in this repo fetched via agents@v0 with explicit FULLSEND_DIR override. It cleanly separates engine/scorer implementation (fullsend) from policy defaults (agents), keeps overrides simple, and matches the ownership model described in the docs.

Documentation (2) +70 / -0

README.mdDocument measurement manifests as online-scoring policy +13/-0

Document measurement manifests as online-scoring policy

• Adds a new section explaining that 'eval/measurements/' defines default online-scoring policy for managed jobs. Clarifies separation from functional eval harness scenarios and links to the companion fullsend PR and ADR.

eval/README.md

README.mdAdd measurements README covering ownership and resolution +57/-0

Add measurements README covering ownership and resolution

• Introduces documentation for what measurement manifests are, why they live with agents, and how managed jobs resolve local overrides vs agents@v0 defaults. Defines responsibility boundaries between fullsend (scorer code) and agents (policy) and lists initial agent coverage for em-001.

eval/measurements/README.md

Other (6) +36 / -0

code.yamlAdd default manifest for code agent (em-001 v1) +6/-0

Add default manifest for code agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'code' agent.

eval/measurements/code.yaml

fix.yamlAdd default manifest for fix agent (em-001 v1) +6/-0

Add default manifest for fix agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'fix' agent.

eval/measurements/fix.yaml

prioritize.yamlAdd default manifest for prioritize agent (em-001 v1) +6/-0

Add default manifest for prioritize agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'prioritize' agent.

eval/measurements/prioritize.yaml

retro.yamlAdd default manifest for retro agent (em-001 v1) +6/-0

Add default manifest for retro agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'retro' agent.

eval/measurements/retro.yaml

review.yamlAdd default manifest for review agent (em-001 v1) +6/-0

Add default manifest for review agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'review' agent.

eval/measurements/review.yaml

triage.yamlAdd default manifest for triage agent (em-001 v1) +6/-0

Add default manifest for triage agent (em-001 v1)

• Adds a per-agent measurement manifest selecting 'trace_fitness' (id em-001) version 1 for the 'triage' agent.

eval/measurements/triage.yaml

Description

The measurement manifests set the trace_fitness measurement id to em-001, but the measurement
README refers to the same measurement as EM-001, creating ambiguity about the canonical identifier
to use in configs/docs/tooling.

Code

eval/measurements/code.yaml[R3-5]

+  - id: em-001
+    scorer: trace_fitness
+    version: 1

Relevance

●●● Strong

Trivial consistency fix; team has accepted casing-alignment changes in nearby review history.

PR-#592

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The repo’s new measurement README explicitly calls the scorer “EM-001” while each new manifest
defines the measurement id as “em-001”, so the PR introduces two different canonical spellings for
the same measurement identifier.

eval/measurements/README.md[10-11]
eval/measurements/code.yaml[1-5]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The measurement ID is referenced with different casing in different newly-added sources: manifests use `em-001`, while documentation calls it `EM-001`. This makes it unclear which identifier is canonical.

## Issue Context
All per-agent manifests added in this PR use `id: em-001`, but the README in the same directory describes the scorer as `EM-001`.

## Fix Focus Areas
- eval/measurements/README.md[10-11]
- eval/measurements/code.yaml[1-5]
- eval/measurements/fix.yaml[1-5]
- eval/measurements/prioritize.yaml[1-5]
- eval/measurements/retro.yaml[1-5]
- eval/measurements/review.yaml[1-5]
- eval/measurements/triage.yaml[1-5]

## Proposed fix
Pick one canonical representation (either `EM-001` everywhere or `em-001` everywhere) and update both:
1) the documentation string in `eval/measurements/README.md`, and
2) all manifest `id:` fields (if needed) to match.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Description

This PR introduces a new measurement-manifest feature and adds 20+ lines of new configuration/docs,
but the provided PR description does not link an authorizing issue. This violates the requirement to
have explicit authorization for non-trivial work.

Code

eval/measurements/README.md[R1-4]

+# Measurement manifests
+
+Per-agent YAML that selects which **eval measurement** scorers run after a
+managed agent job (`fullsend eval-measure`). This is **not** the functional

Relevance

● Weak

Similar “add explicit authorization/linked issue” governance suggestions were rejected previously.

PR-#476
PR-#631

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Rule 1538390 requires an explicit linked issue for non-trivial changes. The diff adds a new
eval/measurements/ feature surface (manifests + documentation), but the provided PR description
contains no linked issue reference (e.g., Fixes #...).

eval/measurements/README.md[1-4]
Skill: pr-review

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The PR introduces a non-trivial new feature/config surface (measurement manifests) but has no linked issue authorizing the work in the PR description.

## Issue Context
Compliance requires non-trivial changes (20+ changed lines or structural changes) to include an explicit linked issue (e.g., `Fixes #123` / `Refs #123`, or a full issue URL) that authorizes the change.

## Fix Focus Areas
- eval/measurements/README.md[1-4]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

💡 Did you know, you can copy the agent prompt from any finding and feed it to your IDE agent

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Review

Findings

High

• [protected-path] .pre-commit-config.yaml — This PR modifies .pre-commit-config.yaml, which is a protected governance/infrastructure file. The PR has no linked issue justifying changes to protected paths. Human approval is required for protected-path changes.
  Remediation: Link an authorizing issue to this PR, or obtain explicit human approval for the protected-path change.

Low

• [redundant validation] eval/lint-measurements.sh:305 — The version field regex in main() overlaps with parse_scalar's validation for present-but-invalid values. The main() check is not dead code — it also catches missing version keys where it.get('version', '') returns ''. The overlap is partial, not total.

• [missing-authorization] No linked GitHub issue for a 769-line, 14-file feature PR. The PR body references companion PR fullsend#6036 and ADR 0087, providing clear context, but non-trivial changes should have explicit authorization via a linked issue per project policy.

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Review

Findings

High

• [protected-path] .pre-commit-config.yaml — This file is under a protected path. The PR has no linked issue in fullsend-ai/agents providing authorization for modifying governance/infrastructure files. The change adds a lint-measurements pre-commit hook entry, which is consistent with the PR's intent and mirrors the existing lint-agent-docs hook pattern, but human approval is always required for protected-path changes regardless of context.
  Remediation: Obtain explicit human approval for this infrastructure change, or link to a tracking issue in fullsend-ai/agents that authorizes the .pre-commit-config.yaml modification.

Low

• [missing-authorization] This PR introduces a non-trivial new feature (measurement manifests, lint tooling, pre-commit hooks, documentation — ~600 lines across 14 files) with no linked issue in the fullsend-ai/agents repository. The PR body references a companion platform PR (fullsend#6036 / ADR 0087), which provides design-level context and adequate authorization signal.
  Remediation: Consider creating a tracking issue in fullsend-ai/agents for traceability.

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Review

Findings

High

• [protected-path] .pre-commit-config.yaml — This PR modifies .pre-commit-config.yaml, a protected governance/infrastructure file. The PR has no linked issue to establish authorization for this change. Human approval is always required for protected-path changes.
  Remediation: Link an authorizing issue to this PR that justifies modifying .pre-commit-config.yaml, or obtain human approval for the protected-path change.

Low

• [naming-convention] eval/lint-measurements.sh:1 — The new pre-commit local hook entry script uses a .sh extension (eval/lint-measurements.sh), while the existing analogous hook script has no extension (hack/lint-agent-docs). Both are language: script hooks serving the same purpose. The codebase convention for pre-commit local-hook linter scripts is extensionless.

• [missing-authorization] No linked issue for this non-trivial feature PR (578 additions, 14 files). The PR body provides context via companion PR feat(eval)!: add eval measurements and EM-001 trace_fitness scorer fullsend#6036 and ADR 0087 but no authorizing issue is linked in this repository for scope traceability.

Labels: PR adds new eval/measurements subsystem (YAML manifests, linter, tests) — feature addition, not purely documentation

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Review

Findings

High

• [protected-path] .pre-commit-config.yaml — This PR modifies .pre-commit-config.yaml, which is a protected governance/infrastructure file. The PR has no linked issue to authorize changes to protected paths. Human approval is always required for protected-path changes.
  Remediation: Ensure a human reviewer explicitly approves the changes to .pre-commit-config.yaml.

Low

• [variable-naming-consistency] eval/lint-measurements.sh:24 — Counter variable naming is inconsistent with eval/lint-cases.sh, the closest sibling linter in the eval/ tree. eval/lint-cases.sh uses uppercase ERRORS=0, while this script uses lowercase errors=0 and file_count=0. Although hack/lint-agent-docs uses lowercase, eval/lint-cases.sh is the more relevant local precedent.

• [error-message-consistency] eval/lint-measurements.sh:179 — Final error message uses failure(s) while the sibling linter eval/lint-cases.sh uses failures without the parenthetical.

• [yaml-document-marker-consistency] eval/measurements/code.yaml:1 — The six new measurement manifests use the YAML document start marker (---) on line 1, while all existing YAML files under eval/ (e.g. eval/code/eval.yaml, eval/triage/cases/001-bug-url-encoding/input.yaml) omit it. This introduces a minor style inconsistency within the eval/ tree.

• [incomplete documentation] LOCAL.md:133 — LOCAL.md describes the eval/ directory as containing only 'functional test scenarios that run agents against ephemeral GitHub repos and score the results.' With this PR, eval/ now also contains eval/measurements/ — default online-scoring measurement manifests consumed by fullsend eval-measure. The description is not wrong but is now incomplete; eval/README.md (updated in this PR) serves as the canonical reference.
  Remediation: Add a brief mention of measurement manifests to the eval/ description in LOCAL.md.

• [missing documentation] README.md:18 — The 'Repository structure' section lists all major directories but omits eval/, a significant top-level directory. While this omission predates this PR, it becomes more noticeable with measurement manifests added.
  Remediation: Add eval/ to the directory listing in README.md.

Next steps:

• /fs-fix — agent addresses review findings automatically
• /fs-fix <your instruction> — agent fixes with your specific guidance
• Push commits directly — review re-runs automatically on push
• /fs-fix-stop — disable automatic fix runs for this PR

Review

Findings

Low

• [yaml-document-marker-consistency] eval/measurements/code.yaml:1 — The six new measurement manifests use the YAML document start marker (---) on line 1, while all existing YAML files under eval/ (e.g. eval/code/eval.yaml, eval/triage/cases/001-bug-url-encoding/input.yaml) omit it. This introduces a minor style inconsistency within the eval/ tree. Since these files live in their own subdirectory and serve a distinct purpose, this is cosmetic rather than harmful, but dropping the --- would align with the established convention.

• [incomplete documentation] LOCAL.md:133 — LOCAL.md describes the eval/ directory as containing only 'functional test scenarios that run agents against ephemeral GitHub repos and score the results.' With this PR, eval/ now also contains eval/measurements/ — default online-scoring measurement manifests consumed by fullsend eval-measure. The description is not wrong but is now incomplete; a reader following this guide would not learn that eval/ also houses measurement manifests.
  Remediation: Add a sentence after line 135 mentioning the measurement manifests, e.g.: 'The eval/measurements/ subdirectory holds default online-scoring manifests (see eval/measurements/README.md).'

Review

Findings

Low

• [documentation-link-consistency] eval/README.md:151 — Reference to 'fullsend ADR 0087' uses plain text while adjacent references in the same sentence use markdown hyperlinks (e.g., [fullsend#6036](...)). The eval/measurements/README.md already hyperlinks the same ADR with a direct URL, so the link target is known and available.

Labels: PR adds YAML policy files and documentation for eval measurement manifests

Review

Findings

Low

• [scope-architectural-inconsistency] eval/measurements/prioritize.yaml — Measurement manifest exists for the 'prioritize' agent, but no functional eval directory exists at eval/prioritize/. The measurements README explains these are separate systems, but an explicit note that measurement coverage is independent of functional eval coverage would prevent future confusion.

• [scope-architectural-inconsistency] eval/measurements/retro.yaml — Same as above for the 'retro' agent — no functional eval directory at eval/retro/. Consider a one-line clarification in the measurements README.

• [scope-missing-dependency] eval/measurements/README.md:14 — References 'fullsend ADR 0087' as the authoritative specification without a direct URL. The ADR lives in the external fullsend-ai/fullsend repository (a standard cross-repo reference pattern), but adding a direct link when the ADR is published would improve discoverability.